Make extract_levels preprocessor function lazy #1761
Conversation
bouweandela
changed the title
Codecov Report
@@ Coverage Diff @@
## main #1761 +/- ##
=======================================
Coverage 92.79% 92.80%
=======================================
Files 236 236
Lines 12485 12490 +5
=======================================
+ Hits 11586 11591 +5
Misses 899 899
|
| def get_array_module(*args): | ||
| """Return the best matching array module. | ||
|
|
||
| If at least one of the arguments is a :class:`dask.array.Array` object, | ||
| the :mod:`dask.array` module is returned. In all other cases the | ||
| :mod:`numpy` module is returned. | ||
| """ | ||
| for arg in args: | ||
| if isinstance(arg, da.Array): | ||
| return da | ||
| return np |
There was a problem hiding this comment.
This function shouldn't be needed thanks to the dispatch mechanism from NEP-18. However, it is needed, which indicates a bug in numpy wrt dispatching in np.ma. If you have the stomach for it, it would be nice to follow up with an issue upstream to sort this out.
There was a problem hiding this comment.
It looks like it's work in progress: numpy/numpy#22913
There was a problem hiding this comment.
maybe add a TODO to remove when that'll be in, and add the link to the Numpy PR?
There was a problem hiding this comment.
Not too keen on that, it will take quite a while before we drop support for the last numpy version that doesn't have the feature.
There was a problem hiding this comment.
good point! We can always pin numpy - but I wouldn't do that
|
Thanks for reviewing! |
|
My pleasure! maybe @zklaus can have a look at this one last time and merge, please 🍺 |
|
@bouweandela I even fixed the conflicts, that's how charitable I feel today 😆 Maybe you can harpoon Klaus do a final review and merge? |
There was a problem hiding this comment.
Thanks for tacking this, @bouweandela! And thanks for the review @valeriupredoi! I've just used this branch to run a bunch of recipes, all seems fine. This PR looks good to me too, merging now 👍
Description
Combined with v0.3 of
stratifythis makes theextract_levelspreprocessor function lazy.Related to #674 and #35.
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
To help with the number pull requests: